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In its role as a cooperating and supporting 
sponsor of the Second Annual Pennsylvania conference on 
Post— Secondary Occupational Education, the Center for 
the Study of Higher Education believes it is being 
highly responsible as well as responsive to the higher 
education co mm unity of the commonwealth. Occupational 
or paraprofessional education, post— secondary in 
character, is of immense significance. The young person 
who enters the labor market today is required more and 
more to have an armamentarium of considerable under- 
standing and skill in a somewhat specialized area. 

The post-secondary institutions have, to some degree, 
stabilized education in the technical areas. They are, 
however, not much beyond the pioneering state in the 
area of human service occupations. Sound analysis, as 
represented in this conference report, can only be 
helpful to those who labor r (« the task of providing 
occupational education. 

The focus of this conference on evaluation 
and effectiveness is also appropriately timely. Higher 
education is today being asked to be accountable as 
it has never been asked to before. This conference 
and this report are themselves evidence of the sen- 
sitivity of higher education to its responsibilities. 

6. Lester Anderson 
June, 1971 
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INTRODUCTION 



Angelo C. Gillie 

Associate Professor, Department of Vocational Education 
The Pennsylvania State University 



I think it would he appropriate to briefly 
review the series of events that culminated in the 
planning of this annual event. The idea of conducting 
a statewide conference on post-secondary occupational 
education for Pennsylvania evolved from several discus- 
sions with Robert Knoebel, who is presently Director 
of the Bureau of Academic Services, Department of 
Education of the Commonwealth of Pennsylvania. Mr. 
Knoebel was Director of the Bureau of Community Colleges 
of the Department of Education at the time. These 
initial meetings took place at the American Association 
of Junior Colleges 1 annual convention in Atlanta in 
March 1969, 

Our original basis for discussions was our 
eonmion belief that The Pennsylvania State University 
and those pos t-second ary institutions that offered 
occupational programs should enter into joint ventures 
that could lead to the overall betterment of post- 
secondary occupational education in Pennsylvania. 

These conversations ranged over a number of possible 
cooperative activities including research studies, 
teacher preparation programs and conferences . There 
has been some progress made in each of these areas, 
but the first one to be acted upon dealt with a state- 
wide conference cn post-secondary occupational education. 

An advisory committee was established in 
early spring of 1969, to delve into the possibility of 
planning the statewide conference. This committee was 
under the leadership of Robert L. Sheppard, Chief of 
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Occupational Education Programs for the Bureau of 
Community Colleges, The committee consisted of educa 
tors from several of the community colleges , the 
Department of Education, and from the main campus of 
the university. 

After considerable debate and planning, the 
first conference was conducted in October 1969, with 
its overall topic, "Post-Secondary Occupational 
Education; An Overview and Strategies," The confer- 
ence received strong support from its inception, IMe 
community colleges and what was then the Bureau of 
Community Colleges in the Department of Education were 
actively involved in the planning. The Department of 
Vocational Education and the Center for the Study of 
Higher Education were the university components 
actively involved in the planning and financing of the 
event, A monograph entitled Post-Secondary Occupational 
Education; An Overview a nd Strategies was one of the 
outputs of the first conference. 

Another outcome, and even more important to 
us in Pennsylvania, was the resultant improvement in 
the relationships between the various institutions in 
the state that offered post-secondary occupational 
education and The Pennsylvania State University. An 
evaluation of that conference led us to believe there 
would be considerable value in conducting a conference 
devoted exclusively to post-secondary occupations 
education on an annual basis. The event reported in 
this monograph is the result of that decision and is 
the second of what we hope will be many annual con- 
ferenees . 

This conference » like the firs t * utilized an 
advisory committee as a planning vehicle,. An effort 
was made to bring together a group of individuals that 
would adequately represent the various post-secondary 
institutions that offer occupational programs in 
Pennsylvania (the committee members are listed in 
Appendix C ). Mr. Sheppard was again asKejJ to serve as 
chairman. It should be noted that in addition to being 
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one of the most knowledgeable persons on post*- 
seeondary occupational education in Pennsylvania, 

Mr. Sheppard has also displayed a remarkable degree 
of skill in utilising the diverse suggestions offered 
by the advisory group. Fiscal support came from the 
sources that sponsored the first one. The Center for 
the Study of Higher Education subsidized the major cost 
of the conference and this publication. Important 
financial support was also made available by the Depart- 
ment of Vocational Education. 

The institutions represented at this con- 
ference were more diverse than those of last year, 
with more involvement by the Commonwealth campuses of 
Penn State, private post-secondary schools and area 
vocational- technical schools* The major objectives of 
the conference were: (1) to provide an authoritative 

presentation on evaluation of post-secondary occupa- 
tional education in terms of programs, faculty, and 
institutions; (2) to provide the participants with 
information that would better enable them to identify 
some elements and useful approaches for evaluation of 
programs, faculty, and their institutions; (3) to pro- 
vide an opportunity for educators concerned with post- 
secondary occupational education to meet for the pur- 
pose of exchanging ideas and viewpoints on evaluation; 
and (4) to continue the series of annual conferences 
on two-year college occupational education as a 
cooperative venture between the university and post- 
secondary institutions that offer occupational programs. 

Our speakers were selected because of their 
expertise in the area of evaluation. Dr. Medsker, a 
leading national figure in the two-year college move- 
ment for many years, provided us with valuable Insight 
into strategies for the evaluation of post-secondary 
occupational programs* He made a distinction between 
the process and the product approach and. drew upon the 
Tyler and Glaser evaluation models. One of the major 
points made by Medsker was about the need for a systems 
approach to the evaluation of occupational education 
(which starts with goal statements, ad ends with 
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evaluation of goal achievements) . His presentation 
also discussed the difficulty in conceptualizing voca- 
tional education in a changing society; ascertaining 
the objectives of occupational programs: and evaluat- 
ing programs designed primarily for retraining purposes. 

He then went on to distinguish between external and 
internal evaluation, pointing out that evaluation by 
state agencies and accreditation of programs will become 
increasingly more important in the years ahead, pre- 
dicting that more accountability will be demanded in 
all aspects of education. Also predicted by Dr. 

Medsker was the emergence of new kinds of worker capa- 
bilities not treated in the present training and 
education programs , The treatment of this topic was 
concluded with reactor presentations by Drs. John L. 
Leathers and Raymond Pietak. 

Dr, Robert Wiegman, also a nationally known 
educator in the two-year college movement, presented 
fc.tie major paper on the topic of faculty evaluation. 

He presented and elaborated on six suggestions for use 
in assessing teacher effectiveness: 1) teacher selection 

interviews should include questions dealing with teach- 
ing ability; 2) evidence of teaching performance 
should be collected on a continuing basis; 3) it 
should be determined whether the teacher has stated his 
course objectives in clear enough terms for the 
students to know what is expected of them; 4) an 
assessment of the product of instruction should be made, 
including a study of the employment pattern of the grad- 
uates, as well as other data derived from well-designed 
and conducted follow-up studies; 5) determination of 
the instructor's service to the college; and 6) a search 
for evidence as to how the instructor provides for input 
from the field should be made. For example, to what 
extent does he utilize advisory committees? How does 
he r ema in up to date in his specialized field? To what 
degree and in what manner does he maintain contact with 
the students outside the classroom? Dr, Wiegman ' s con- 
cluding statement went to the heart of the matter: If we 

wish to improve instruction in cur colleges the first order 
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of business should be that we focus on that topic." 

The treatment of this topic was rounded out by the 
presentations of Dr. Eisenstein and Mr, Kray. 

Dr. Salatino dealt with the problem of 
evaluation of institutions, the third topic of the 
conference. He listed several types of contemporary 
approaches that are being used for purposes of account- 
ability in education including the program, planning, 
and budgeting system; the national assessment of educa- 
tional progress; performance contract programs with 
private concerns; state program audits; and audits at 
the federal level. Dr, Morrissey and Mr, Snyder 
followed with presentations in response to Dr. Salatino* s 
paper , 



Upon reviewing the presentations made on the 
three topics (evaluation of programs, faculty, and 
institutions in post-secondary occupational education) , 
it appears that they have been well covered. The extent 
to which the conference participants agreed with this 
conclusion is dealt with in the evaluation section of 
this publication. 

The editor wishes to express his appreciation 
for the financial assistance received from the Center 
for the Study of Higher Education and Dr. G, Lester 
Anderson, its director. Special thanks are also offered 
to Richard Olson and Cheryl Gumaelius for their help 
during and after the conference , and to Sharon M. 
Friedman who supervised the final preparation of this 
manuscript . 
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STRATEGIES FOR EVALUATION OF 
POST-SECONDARY OCCUPATIONAL PROGRAMS 



Leland L. Medsker 

Director * Center for Research and Development 
in Higher Education 
University of California at Berkeley 



As one considers the subject of appropriate 
strategies for the evaluation of post-secondary voca- 
tional programs, he is confronted at once by the 
magnitude and difficulty of the problem* The magnitude 
is great, due in part to the rapid growth of occupational 
education* In the four years from 1965 to 1969, federal 
subsidies to vocational programs increased from $605 
million to $1*4 billion. Enrollments in the same four- 
year period increased 46 percent over those reported 
in 1964* Post-high school vocational education shbwed 
the largest increase— 19 percent. The number of voca- 
tional education teachers at the post-secondary level 
increased by 39 percent. ^ If one adds to these 

figures comparable data for technical occupational pro^- 
grams in community colleges and other types of post- 
secondary institutions that are supported entirely by 
local funds, the growth factor Is even more astounding. 
There is also further documentation that their growth 



^Data reported in ECS Bulletin (September , 



1970) 



is occurring most rapidly at the post-secondary level. 
Furthermore, it is likely that growth will continue to 
be encouraged in high places. Representst- ~ Edith 
Green said in an address last summer at the annual meet 
ing of the Education Commission of the States: 

Then in the 1970s let us adopt a new Magna 
Carta for the vocational and technical 
occupations and an educational system to 
enforce that recognition. In Congressional 
action, I predict increased financial aid 
to this fastest growing section— our com- 
munity colleges and our technical institutes. 

Sheer growth naturally implies difficulty in 
evaluation simply because there is so much to evaluate. 
But there are qualitative as well as quantitative prob- 
lems in evaluation. Curriculum evaluation at any level, 
or pertaining to any subject, is inherently difficult 
and although much has been written about it , a review 
of the literature suggests that it is the one area of 
education in which the least progress has been made. 

At first blush we might conclude that the evaluation of 
occupational programs would be far simpler than the 
assessment of academic programs , since its product can 
be measured more easily. However, as I shall indicate 
later, only minimal efforts to measure the product of 
occupational programs have been made. Furthermore, 
vocational education is a many-faceted, multi-dimensional 
effort that is not easily compartmentalized into neat 
packages for assessment purposes. 

Perhaps to no one’s surprise, a rather thorough 
search of the literature reveals a paucity of information 
about the evaluation of occupational education. We are 
indebted to Kaufman.^ for his comprehensive treatment of 



2 J. J. Kaufman, et al . The Preparation of Youth 
for Effective Occupational Utilization, The Role of the 
Secondary School in the Preparation of Youth for Employ - 
ment (University Park; Pennsylvania State University, 1967). 
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the subject in secondary schools, although he does not 
deal extensively with strategies. Numerous individual 
studies of vocational programs (mostly at the local 
level) are reported through the ERIC Clearinghouse for 
Higher Education and dissertation abstracts, but gen- 
erally there is a dearth of comprehensive treatments of 
the subject. 

As I scanned the literature and pondered the 
subject, I concluded that it takes a venturesome indi- 
vidual to discuss strategies, but nevertheless, it is a 
worthwhile task to undertake. I deal with the topic 
not as a vocational educator , but simply as one concerned 
with realistic educational programs in post-secondary 
institutions. In the remarks to follow, I first discuss, 
within the context of evaluative procedures in general, 
some of the approaches and techniques for evaluating 
these vocational programs. This is followed by some 
comments about certain problems and issues that arise 
in connection with such evaluation and 1 conclude with 
a few remarks concerning possible directions for the 
future. 

You will observe that my plea is for a more 
thorough, systematic approach to evaluation— an approacn 
that places primary emphasis on product instead of on 
process. I divulge this now so that ray position is 
cleat; from the outset* 

Approaches and Techniques 

Let us turn first to a discussion of what 
appear to be the two principal approaches to the eval- 
uation of occupational programs, namely, the process 
approach and the product approach. The process method 
is one that is largely concerned with how the program 
functions, including the various environmental elements 
that presumably determine the degree of its success. 

This approach would evaluate such factors as the cur- 
riculum and how it is developed , the use of advisory 
mechanisms, the equipment used for instructional pur- 
poses, the quality of the faculty, the methods of teach- 
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ing, the process of selecting students, and the manner 
in which placements are made, to name only a few. The 
premise is that by an evaluation of program character- 
istics, conclusions regarding the viability of the 
training can be inferred. 

The product approach is based on concern for 
the student and what the training does for him. It 
begins with attention to program objectives and ends 
with an inquiry into program outcomes, both qualitative 
and quantitative. It is particularly concerned with 
how well the program fulfills its objectives; the 
extent to which students persist, how they find jobs 
appropriate to their training, and h-w they perform in 
these jobs. Naturally, there are other overall prod- 
uct concerns, an important one being the extent to 
which the training program meets local, state, and 
national manpower demands . 

Differences between the two approaches are 
readily apparent. One is descriptive, the other attempts 
to measure performance. One asks how it is done, the 
other asks how well it is done. 

An obvious question is whether the two 
approaches are mutually exclusive. An equally Obvious 
answer is that they are not. In fact, given the 
assumption that program characteristics are related to 
outcomes, one must deduce that both approaches are 
important. This is so to the extent that characteristics 
partially explain the level of success that a program 
enjoys. Thus, they constitute independent variables 
in accounting for various degrees of performance. This 
concept will be returned to later. 

A much more significant question is whether 
either of the two approaches is more important than the 
other~or at least whether either should constitute the 
basic approach. This issue must be examined within 
the framework of existing concepts about educational 
evaluation in general. Naturally it is impossible to 
cover here the many emerging ideas concerning how best 
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to evaluate educational programs. Many of them date 
back to the model originally advocated by Ralph Tyler 
in the 1930s , in which he stressed the development of 
educational objectives stated in operational terms so 
that data could be collected to determine how well the 
objectives are achieved. The idea subsequently was 
further advanced by Gagne and by many others. Writing 
in the Spring 1969 issue of the Journal of Trade and 
Industrial Teacher Education » J, Thomas Hastings, 
director of the Center for Instructional Research and 
Curriculum Evaluation at the University of Illinois, 
dwelt at some length on an evaluation model developed 
by Stake as depicted in Figure 1, As pointed out by 
Hastings, the schema, while placing heavy emphasis on 
outcomes , also emphasizes other aspects of the educa- 
tional program. The rows represent three aspects of 
the program. By "antecedents , " it is meant those 
aspects of the situation that come before but are 
highly relevant to the instructional program under con- 
sideration, They include such things as student 
characteristics, the attitudes and background of 
faculty, and the general school environment. 
'’Transactions" mean the operational aspects of the 
actual program. The "outcomes" row refers not only to 
student outcomes but also to such outcomes as the Impact 
of the program on the institution itself and on its 
constituency. The four columns of the matrix are des- 
cribed as the sources of data for evaluation, "Intents" 
refers to expectations expressed by students, parents, 
employers and others. The column headed by 
"observations" refers to expectations gleaned from 
tests, interviews, and other techniques. "Standards" 
implies the collection of information related to 
expected performance levels in such matters as admis- 
sions, operational modes and final student outcomes. 

The final column suggests the possibility of collect- 
ing data on the value judgments and the people or 
groups who make them. Of this Hastings saidi 

There are various value orders eve^ where 
the values themselves are agreed upon as 
generally important. In collecting our 
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figure 1 
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*r. e. Stake, "The Countenance of Educational 
Evaluation," Teachers College Record , 68 (1967) , 523-540 
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data at any of the levels (from 
Congressional committee through the 
State Department of Public Instruc- 
tion to the teacher) , consideration 
must be given to describing the kinds 
of judgments which are made* The 
question , "What kinds of groups make 
what kinds of judgments?" is one that 
we have too often left out of the 
evaluative process . In many cases 
we have missed by that amount an 
optimal movement toward rational 
decisionmaking [ sic] • 

I have described the Stake model as discussed 
by Hastings only to indicate the many variables and 
their interrelationships that may well be considered in 
the evaluative process. In fact* this is the primary 
value of any model regardless of its purpose* 

A more simple and direct evaluation approach 
has been described by Robert Glaser^ in which he 
suggests the following sequence of operations: 

1* Outcomes of learning are specified in 
terms of the behavioral manifestations 
of competence and the conditions under 
which it is to be exercised. 

2, Detailed diagnosis is made of the 

Initial state of a learner coming into 
a particular instructional situation. 

This careful workup of student perfor- 
mance characteristics relevant to the 
instruction at hand is necessary to 



3 R. Glaser, Evaluation of Instruction and 
Changing Educational Models (Los Angeles: Center for 

the Study of Evaluation of Instructional Programs , 
University of California, 1968) • 
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pursue further education. 

The immediate instructional step consists 
of educational alternatives adaptive to 
the classifications resulting from the 
initial student educational profiles, 

4. As the student learns, his performance is 
monitored and continuously assessed at 
longer or shorter intervals appropriate 
to what is being taught. 

5 Instruction and learning proceed in a 

cybernetic fashion, tracking the perfor- 
mance and selections of the student. 
Assessment and performance are interlinked, 
one determining the nature and requirement 
for the other, 

6. Inherent in the system’s design is its 

capability for improving itself. Perhaps 
a major defect in the implementation of 
educational innovations, especially in 
the area of individualization, has been 
the lack of the cumulative attainment of 
knowledge — on the basis of which the next 
innovation is better than the one that 
preceded it* 



Despite the advocacy of such models as those 
jst discussed, the literature confirms the fact that 
d date the principal emphasis has been on process 
ather than on product evaluation. One has only to ®can 
he criteria used for evaluation by innumerable schools 
nd agencies to realize this fact, Norman Harris, 
riting in 1964, grouped a long list of important guide 
ines for evaluation into the following program 



4 n q Harris, Technical Education in the 
T„r>-tm- College (Washington, D.G, ; American Association 

of Junior Colleges , 1964) . 
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categories : 1) meeting community needs; 2) adminis- 

trative "climate 11 and organization* 3) guidance place- 
ment and follow-up ; 4) instruction and curriculum; and 
5) facilities and equipment. Even in the accreditation 
of vocational education this is the tendency, as John 
Stanavage of the North Central Association indicated 
at a recent meeting in Chicago when he said, "We never 
go into a school without trying to evaluate each* pro- 
gram. But I would have to concede that we are looking 
at process and that we can’t be confident of the 
validity of evaluation." 

Implicit in Br, Stanavage f s statement and in 
those of many other individuals is the opinion that 
evaluation should be based more on product than process. 
This opinion was also expressed by Moss when he wrote: 

One of the most critical aspects of 
program evaluation, and the one which 
has thus far probably received the 
least attention, is the identification 
and measurement of the program outcomes 
which are to serve as evaluative criteria. 
Everyone affected by evaluation, and that 
is all educators, must be concerned with 
developing as complete an array of 
relevant potential outcomes as possible 
for use by evaluators . 5 

Note that Moss refers to two components of evaluation: 
the identification of program outcomes (objectives) 
and the measurement of them. He later states that 
evaluative criteria should be in terms of outcomes 



“\j. Moss, Jr. Review of Research in 
Vocational-Technical Teacher Education (Minneapolis : 
Research Coordinating Unit in Occupational Education, 
1967) . 
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Instead of program characteristics and suggests 
”l masL including formative evaluation expert 
“d self-evaluations , f„llo»-upa. experiments, inter 
rupted time series, and regresslonal analysis. 



The most dear-out conclusion '“le drawn 
i r "h« e there e L 0 Led%ora°s%“M“pproac!l to the 

dot^collect todays tem^that 3 sheds light on innumerable 

vmriablea*and nidi in X 

^e“:r t :vdLdor:rfe'c q onduuLr n au r d P rogr r 

lan ll mdlded Such a plan ^4^ 

approach but does not neglect the program a^oach. 



A systems approach for use at the state 
stresses the lattar* 



Figure 2 (pages 18-19) portrays the substance 
- the ulan. The fact that the evaluation program 
starts with the formulation of goal statements and ends 
with* the formulation of goal statements f” d ends h 

s SSSSSrSSSrsSs- 

strategies for achieving them. 



Certain operational characteristics of the 
desired system are set forth as follows; 



6 l Sgtg for S e en Evaluation of YucsclpHal 
nauctlon. toll jjaegr t. RescarchSeries no. so. -dii 
23S (The Ohio State University, 19/ / * 
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1* Information which stresses program out- 
comes rather than processes . 

2. Measurement of the effectiveness of efforts 
in achieving projected program goals. 

3. Sys tematic feedback of relevant evaluative 
information to local schools. 

4. Sufficient flexibility to permit individ- 
ual states to modify system components 
without impairing the integrity of the 
system. 

5. Articulation with other program planning 
systems, including PPB3 and manpower. 

The group that developed the plan commented 
on two evaluation methods now in common use by state 
divisions, namely, process evaluation and all- 
inclusive data banks. Both were rejected in favor of 
a plan that would be compatible with the systems 
approach. The plan provides for data collection from 
the various types of vocational schools in the state 
and, as indicated above, includes data on the program 
characteristics in the schools and information gleaned 
from the follow-up of students* Those involved in the 
project articulated the problem of evaluation as 
follows i 



Although it is clear that the evalu- 
ation of program adequacy is an 
essential part of the assessment of 
a state program, it also is true that 
the value and effectiveness of a pro- 
gram can be determined only with 
respect to outcomes. Thus, a pro- 
gram which is effective for objective 
X may be ineffective for objective Y. 
A facility, a teaching staff, or a 
budget is appropriate, adequate, or 
effective only with respect to some 
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Goals Finalized 



goal. Thus, evaluation of educational 
program effectiveness is meaningful 
only after it is known whether educa- 
tional programs served their intended 
purposes , 

This point is being emphasized strongly 
in the political and legislative arenas. 

Not only is federal legislation written 
to emphasize educational outcomes for 
specified groups of students , but edu- 
cation now finds itself having to com- 
pete at all governmental levels with 
other agencies and institutions for 
limited human and economic resources , 
Decisions by policy-making bodies 
regarding resource allocations are 
being made with increasing frequency of 
evidence of program effectiveness, 
relevance to social and economic con- 
ditions, and the degree to which pro- 
grams reflect community, state and 
federal interests and concerns, in such 
an environment, evaluation methodologies 
which have been commonly employed in 
vocational education fail to provide 
the evidence required by policy-making 
bodies and must be replaced by a more 
effective evaluation methodology if a 
proper case is to be made for support. 

I realize that an emphasis on the evaluation 
of product is not a new thought and that despite the 
long-term tendency of evaluators to look at program 
characteristics, there are many instances of attempts 
to find out how well the program functions in terms 
of performance. Examples of follow-up studies abound. 
My concern is that there are too few instances of 
planned evaluation that start with specific goals and 
objectives — even differentiated by targeted student 
groups— and then proceed step by step to systematically 
measure competencies developed during the program and 
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performance on a job after placement. Included in the 
examination should be the relationship between manpower 
needs and the program as measured both by up-to-date 
occupational information on the one hand and the record 
of placements on the other. The evaluative process is 
a research effort that should be continuous, deliberate, 
and programmatic. Obviously, the evaluation program 
should be a cooperative effort involving faculty, 
administrators, and perhaps others on the staff with 
some one person or agency responsible for planning and 
executing it. Whether it be at the local or state 
le^l, it cannot be left to chance. 



It seems, therefore, as if a new day in the 
evaluation of occupational education has dawned and 
that it is now incumbent on everyone involved to con- 
sider a systematic approach to the problem. One may 
also predict that with new methods of evaluating out- 
comes , and the ability to build data banks, we are 
closer to the day when cost benefit analyses can be 
made of our occupational training programs . Jacob 
Kaufman' has recently written on this point. 

A project that should eventually yield 
assistance in making such analyses is the WICHE-MIS 
(Management Information Systems) program. It is also 
significant that vocational education is now one of five 
targeted development and research areas of the U. S • 
Office of Education. Beginning in fiscal year 1972 and 
for four years thereafter, the National Center for 
Educational Research and Development (NCERD) plans to 
fund a number of projects in program planning, manage- 
ment, and evaluation systems and techniques as they 
apply to vocational education. 



J. Kaufman, "Cost-Effectiveness Analysis 
as a Method for the Evaluation of Vocational and 
Technical Education," Journal of Industrial Education , 
6 (1969). 
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Problems and Issues Pertaining to Evaluation 
o£ Occupational Education 

A number of problems and issues bear on the 
evaluation of vocational education. The first relates 
to the very nature of vocational education and to the 
difficulties of conceptualizing it in a changing 
society. As a nation we have a great diversity of 
opinion about what occupational training really is or 
should be. There are disagreements as to its content 
and the degree of specificity that should characterize 
it. We vary tremendously in our idssa auuUt ucV^-Lu-p- 
ment of skills versus the development of general com- 
petencies. We differ in our thoughts on the optimum 
length of training programs. We worry in varying degrees 
about the problems of occupational obsolescence. These 
common concerns are relevant to this discussion in that 
if evaluation systems are to be devised, we have to be 
fairly sure of what it is we should be evaluating. Pre- 
sumably, each institution is privileged to conceive the 
desired nature of its own program and to evaluate 
accordingly, but it must make sure that the evaluation 
strategies consider the rationale that led to its pro- 
gram characteristics. 

A related problem centers around the diffi- 
culties in ascertaining the objectives of occupational 
programs. It is easy to talk about goals, but consid- 
erably more difficult to articulate them. Furthermore, 
there is always a question of whether occupational 
programs should be designed primarily to prepare workers 
for entry jobs or, once the individual is employed, for 
advancement in the world of work. To the extent that 
the institution makes a distinction between these two 
possibilities, so then must this ba clear in the way 
objectives arc stated and the evaluation strategies are 
planned. It is obvious that goals must be specific and 
complete; otherwise, they cannot be measured. Yet 
Glaser has said that If such outcomes as complex reason- 
ing and open— end edness are desirable aspects of human 
behavior, they need to be recognized as assessable 
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goals. And such outcomes may indeed be important even 
in vocational education!^ 

Perhaps one of the most difficult tasks 
encountered in ascertaining objectives is that of artic- 
ulating differences in potential outcomes among students 
with varying characteristics. With more frequency, the 
literature and the legislation pertaining to vocational 
education make references to targeted groups of students. 
We are all aware that post-secondary institutions are 
being called upon to accommodate greater numbers of 
M new l! students, who are of widely varying backgrounds 
in respect to age, socio-economic groups, motivations, 
and abilities. As a result, federal and state govern- 
ments will probably increasingly insist on quite 
specific objectives for special targeted groups of 
students. To the extent that this is so, there are 
obvious implications for evaluation techniques, 

A third problem of a different order is that 
of evaluating programs designed primarily for retrain- 
ing purposes. If all the prognostications concerning 
job obsolescence come true, and if the average worker 
is called upon to change his way of making a living 
several times over the period of his gainful employment — 
or at least periodically to upgrade his skills — then 
occupational training programs will increasingly be 
involved with retraining as opposed to initial pre- 
employment training. Here the task of evaluation will 
be even more difficult since both the objectives and 
the trainees will vary greatly. Many institutions will 
be involved with both pre-employment and retraining, 
thus their system for evaluation will become more com- 
plicated. 



Several references have been made here to the 
importance of evaluating a program in terms of how it 



8 Ibid. 
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meets manpower requirements . Related to this is the 
extent to which it meets other social needs, especial- 
ly whether it attracts and serves a requisite number 
of students from targeted subgroups of the population. 
Such a criterion is becoming more important. A pro- 
gram may be assessed in terms of how well it serves 
its enrolled students but obviously if students who 
should be enrolled are not attracted to it, a link is 
missing. Thus any system must take this factor into 
consideration as it assesses its total program. 

External Evaluation 

So far we have talked mostly about strat- 
egies for internal evaluation. Our discussion, 
however, would be grossly incomplete without mentioning 
external evaluation. Legislation already enacted man- 
dates an increase in evaluation activities by state 
agencies responsible for federal and state funded pro- 
grams. It would seem that close working relationships 
between state and local bodies could result in the pro- 
verbial "killing of two birds with one stone" whereby 
many of the data needed by the institution could be part 
of a state's effort to evaluate its program based on a 
systems approach. 

Another and more complex problem related to 
external evaluation is found in the accre ditation of 
occupational programs. Everyone concerned with occu- 
pational education at the post— high school level is 
familiar with the problems of accreditation. The fact 
that institutional eligibility for receiving federal 
aid for occupational programs or for serving students 
who are financially assisted by the Federal government 
is based on accreditation, makes such problems as the 
following all that much greater: 

1. The wide discrepancy in criteria used 
by ’regional associations to evaluate 
occupational training. 
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2 . The fact that not all post- 

secondary institutions are eligible 
for accreditation by regional 
associations * 

3# The growing involvement of various 
professional associations the 
accreditation of specific occupational 
programs with the consequent confusion 
between institutional and program 
accreditation. 

These and other problems were discussed at 
a meeting last spring by the Education Commission of 
the States, where attention was called to a need for 
greater flexibility in the accreditation of occupational 
programs and to an increasing emphasis on their results * 
More than a year ago, the National Commission on 
Accrediting created an interim council on accreditation 
of occupational and cpecialized education. In May 1969, 
this council proposed that each of the regional asso- 
ciations establish a separate commission for the 
accreditation of such education. The regional asso- 
ciations tended to resist the suggestion, although 
vocational educators generally seemed to favor it. Thus, 
at this time, it appears that there is much yet to be 
accomplished before the regional associations have per- 
fected a viable means of accrediting this segment of 
education. 



Moreover the controversy between the 
regional associations and the various professional 
organizations that accredit individual programs is far 
from settled. In a study of this problem conducted 
at the Berkeley Center a few years ago, we came to the 
conclusion that in many ways such agencies serve a use- 
ful purpose. In our final report we said: 

There exists a great need to devise a 
methodology which allows each to facil- 
itate the other. Perhaps one hope lies 
in the concept of cooperative activity. 
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whieh would give institutions the 
opportunity to derive the benefits of 
assistance and support from teams of 
professionals supplied by the special- 
ised agencies, while maintaining a 
total institutional commitment under 
the supervision of peers representing 
the regional association. 

There should be ways by which institutional 
evaluation of vocational programs and accrediting 
efforts by various outside agencies can complement each 
other. As duplication of effort is costly in time 
and money, it should be the goal of all involved to 
work toward that end. 

Possible Directions Ahead 

Assuming that technical vocational education 
at the post-secondary level is to remain an important 
element in the American educational system, we can 
speculate on some possible directions the evaluation 
of it will take in the years immediately ahead. 

As expressed and implied in these remarks 
thus far, there is almost certain to be a move toward 
highly organized systematic approaches to evaluating 
occupational training. To develop such a system, or 
to modify an existing one for local use, would seem to 
be an exciting venture. It would provide an opportunity 
to use new and modem techniques in assessing both 
inputs and outputs so that evaluation would be contin- 
uous and change could be based on objective data. 



®L. E. Messersmith, & L. L. Medsker, 
Accreditation of Vocational-Technical Curr icula in 
Post-secondary Institutions, (Berkeley : Center for 

Research and Development In Higher Education, 1969). 
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What are some other probable directions? 

For one thing, like all other facets of education, 
occupational training will increasingly be confronted 
with the concept of accountability* Despite the 
current overuse of the term, the emerging notion is 
that schools and colleges must become more responsible 
for fulfilling the explicit and implicit missions they 
undertake. The question arises as to whom accounts 
ability is owed and while, generally speaking, educa- 
tional institutions are responsible to the people who 
support and control them, their residual or basic 
responsibility is to the students who attend them# 
Accountability in education is difficult to define 
but after a thorough discussion of it, Morphetl® et al 
says ''Accountability thus must now be defined as an 
assignable, measurable responsibility to be fulfilled 
under certain conditions and within certain constraints." 
Since the proof of the training is in what stxidents 
learn, we are likely to see a move on the part of mature 
students and the public to look critically at occupa- 
tional training ventures that do a poor job at what 
they purport to do# 

Such a trend could even go so far as to 
establish contracts with students or certain agencies 
to deliver • . « or else. Increasingly, this procedure is 
being subjected to experimentation# The area of voca- 
tional education may prove to be a likely field in 
which this practice will make headway, especially since 
so many proprietary industrial— type organizations are 
offering vocational training. 

A possibility of a different order is that 
emerging manpower demands will include a new list of 



Society," 
ed. by E* 



^ ‘'Evaluating Education in a Changing 
Emerging State Responsibilities for Education , 
L. Morphet ( 1970 ) . 
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worker capabilities over and beyond the skills that are 
commonly the targets o£ training. This is not a new 
thought, of course. For years, the schools have pon 
dared on how they could develop such abstract qualitie 
as initiative, adaptability, loyalty, inquisitiveness 
and many other such attributes. But the period ahead 
will be subject to great social change and it may be 
that development of the individual as a person will 
become as much a part of vocational training as his 
manual or professional skills. To the extent that this 
is so, evaluative techniques will need to be modified 
to measure more subjective outcomes. And this will 
not be easy. 

The list of possible future directions could 
be expanded indefinitely. But even those we have 
mentioned portray the need for an imaginative approach 
to evaluation of the type with which we are concerned 
today. If, up until now, the assessment and updating 
of occupational training at the post-secondary level 
has been somewhat haphazard, a new direction is man 
dated by a changing society. A direction in which 
resource allocations are based more on demonstrated 
effectiveness than on subjective judgment and in which 
the expectations of the beneficiaries of education— 
both the public and the students— are reckoned with as 
never before* 



ERIC 
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COMMENTARY ON DR. MEDSKER* S PAPER 



John Leathers 
Director* Altoona Campus 
The Pennsylvania State University 



I believe we must ask ourselves* evaluation 
within what context? 

The "summons to accountability , M as WXCHE^ 
teamed it, being served on American higher education 
is affecting all segments of education, including post- 
secondary occupational education. Accountability 
demands selectivity and selectivity requires the clar- 
ification of institutional purposes and an ordering of 
priorities. No institution can be all things to all 
people nor should it try or want to be. 

It seems to me the strategy for evaluation 
for any one institution should begin and end within 
the context of that institution's self-professed 
functions and the roles it purports to play. The con- 
stant clarification of institutional purpose and policy 
must precede any intelligent allocation of resources. 



^Ben Lawrence, George Weathersby, and 
Virginia Patterson , eds . , The Outputs of Higher Educa- 
tion : Their Identification/ Measurement, and Evaluation 

(Boulder, Colorado; Western Interstate Commission for 
Higher Education, July 1970), p, 2. 
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With this premise, a framework for evaluation 
could evolve as follows* 

1. You must have a clear understanding 
and devotion to the defined functions 
and avowed purposes of your institution* 
When you appoint a new professor you 
frequently "hire" a new curriculum. 

When vou appoint a new administrator , 
you may "hire" a new charter. Constant 
clarification of institutional purposes 
is a must., 

2. You must develop and maintain a keen 
sense of awareness of the roles your 
institution expects to play and also be 
sensitive to the roles others may expect 
it to play, 

3. Ask yourself the following questions : 

a. What needs to be done? 

This will often require the percep- 
tion of needs not yet apparent to 
the general public. There should be 
a delicate blending between follow- 
ing the market and creating new man- 
power needs . 

b* What are we going to try to do and 
on what level? 

The designation post-secondary occu- 
pational education means different 
things to different people at differ- 
ent times. For example, does post- 
secondary infer that the program is 
offered chronologically after high 
school or does it mean that the level 
of difficulty is beyond high school? 

c. How should we do it? 



4 . 



Do it ! 



a. Are we actually doing what we set 
out to do? 

b, Are we doing it well? 

c. Can we do it better? 

d, What needs to be done? 

This of course leads you back to further con- 
sideration of the institution’s functions and roles, 
and completes the cycle of the procedure to be continu- 
ously repeated. 

The so-called "process inputs” and "product 
outputs" are so interwoven that it is apt to be a 
futile exercise to attempt to isolate one ‘from the 
other. The great debate about the chicken and the egg 
may go on forever but there can be little doubt that 
institutional process precedes institutional product. 

If the process is soundly conceived and deliberately 
implemented the chances that a good product will evolve 
are greatly enhanced. However, we must also be mind- 
ful that human variables are such that an individual 
"good product” might happen in spite of the process 
and not because of it. 

"Product" in its earliest stages need not be 
considered external to the process* The product begins 
to emerge when the first word is spoken in the very 
first class. Consequently, evaluation of the product 
should also begin then and not be postponed until the 
student is placed in a praeticum situation or on his 
first job. In ray judgment, it is unthinkable that a 
program could be implemented without deliberately 
planning and allocating funds required to carefully 
evaluate the student in the praeticum and the graduate 
on the job. 
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In addition to those references discussed by 
Dr, Medsker, there are a few other sources that come 
to mind as being worth checking, especially to the new 
comers to the field such as I. The American Society 
for Engineering Education has provided effective pro 
fessional guidance for years in the evaluation of 
technical curricula. Their 1962 edition, character- 
istics of Excellence i n Engineering T echnology Educa- 
tion sets forth a good basic approach that with 
^iTTslon could possibly be useful in other disciplines. 
It was helpful to me, for example, just to be reminded 
that the level of a program is determined by its 
objectives, and the quality by how well it achieves 
these objectives. They also very simply point out 
that some technical curricula intend to cover materials 
at a level very near to that of an engineering 
Sd others at approximately the seme level of difficult, 
as the secondary school. The point of my emphasizing 
this now is that each of us must determine the level 
of our programs early in the planning stage and cer- 
tainly before the process inputs are considered. 

The 1969 report of the Engineering Council 
for Professional Development 3 is a current reporting 
of that organization's efforts concerning evaluation, 
accreditation and so on. Also, the Technical Institute 
Division of the University of Texas at Arlington has 
produced a one-page illustration of an engineering 



I -American Society for Engineering Education, 

I Characteristics of Excellence in Engine ering Technology 

£ Education (1962 j , pp. li-U. 

r - - 

| Engineer's Council for Professional Develop- 

| ment, 37th Annual Report for the Year Ending Septembe r 

I 30. 1969 (New York). 

fe- ■ " 

r ■ 

S ^Technical Institute Division, University of 

I Texas at Arlington, Occupational Spectrum. 

f 
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occupational spectrum ranging from the practical to the 
theoretical . It also highlights the wisdom of an 
early decision regarding the level of difficulty of 
each program. 

One othei resource that should be mentioned 
is the 1970 publication by WIGHE, The Outputs of 
H i gh e r Educa 1 1 on » 5 which is certainly one of the latest 
and most comprehensive treatments of evaluation within 
higher education. Dr. Medsker is a cor "ributor to that 
publication. 

If we are going to fulfill the missions of 
our institutions effectively, efficiently and with 
pride, we must evaluate ourselves and our programs 
critically and continuously • Evaluation procedures 
taken out of perspective can become a sort of mystic 
all of its owru Evaluation cannot be permitted to 
become an end in itself, but rather it must serve as a 
means to the end of better teaching, an improved pro- 
cess and products of the highest quality. 



Education. 



■’Lawrence et al* eds . 



The Outputs of Higher 
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COMMENTARY ON DR. MEDSKER'S PAPER 



Raymond A. Fietak 

Provost j Community College of Philadelphia 



Introduction 



Dr, Medsker's provocative paper does compel 
ne to carefully analyze his own philosophical position 
oncerning the evaluation of occupational programs. He 
ia s most certainly incorporated in his paper many of 
he significant elements of this critical and timely 
ub jeet in education. 

As a reactor, 1 will exercise my prerogative 
>f reacting to what interests me in Dr. Medsker s 
>aper. As a matter of fact, I do plan to introduce 
jther information, hopefully, in an attempt to rein- 
force what he has said and to cast further lignt on 
this most interesting and challenging area. 

My remarks will be divided into at least 
three areas and represent the thoughts of an adminis 
trator who must be considered a generalist as opposed 
to being a technical or occupational specialist. The 
first area will concern itself with a conceptual or 
theoretical approach to evaluation. The second are. 
will briefly describe our own attempts at the Community 
College of Philadelphia to systematize the ongoing 
evaluation of occupational programs. I would cate 
gorize our own attempts as being sincere, interesting, 
highly educational and time-demanding! I would not 
say they are all-encompassing in terms of the mode 
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that I will initially describe although many of the 
features of this model are present. 

The third area will briefly review some of 
the evaluation techniques that now exist and are used 
by many of us in our work. They are offered to merely 
illustrate what the literature has been describing 
recently and to point out that evaluation has been 
taking place in occupational education. Judgment as 
to their efficacy will not be rendered. This Must be 
individually done and based on one's own individual 
experiences . 

Evaluation Theory 

Dr. Medsker's plea for a more thorough, 
systematic approach to evaluation with the primary 
emphasis on "product" instead of on "process" and his 
description of several models leads me to observe that 
one of the questions in education that is still per- 
plexing and challenging many of us is the development 
of a theory of evaluation that would! 

1) provide a conceptual scheme for classify- 
ing evaluation problems or areas; 

2) identify the data, means of analysis and 
reporting procedures that may be used; 

3) provide a system of guidelines that would 
assist us in determining what evaluation 
procedures and techniques should be used 
in certain cases . 

Ideally the propositions presented in a 
theory of evaluation should enable us to predict the 
appropriateness of utilizing various evaluation strat- 
egies within a system. Fortunately, the Center for the 
Study of Evaluation at UCLA has made some initial pro- 
gress in this direction. I would like to present some 
of their thinking to you, since it seems to me that 
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their theory development has incorporated much of what 
Dr. Medsker considers desirable.^ 

They prefer the following definition of 
evaluation ; 

Evaluation is the process of ascertaining 
the decision areas of concern, selecting 
appropriate information, and collecting 
and analyzing information in order to 
report summary data useful to decision- 
makers in selecting among alternatives.^ 

Their definition of evaluation is based on 
the following assumptions: 

1. Evaluation is a process of gathering 
information. It is their contention 
that past definitions of evaluation 
were inadequate since they did not 
cover the full range of activities 
requiring information. 

2, Information collected in an evaluation 
will be used mainly to make decisions 
about alternative courses of action. 

Thus the method of information collection 
and the analysis procedures must meet the 
needs of those involved in the decision 
process . 



^The evaluation theory presented in this paper 
is based on work completed at the Center for the Study 
of Evaluation at UCLA and published in an article by 
Marvin C. Alkin, "Evaluation Theory Development , n UCLA 
Eva luati on C ommgnt , 2, 1 (October, 1969), 2—7. 



2 Ibid . , p, 2. 
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3* The presentation of evaluation infor- 
mation to the decision maker should be 
in a form that is highly understandable, 

4 t The evaluation procedures utilized will 
depend on the decisions required. 



I 

4 



O 
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The center staff maintain that evaluation 
must take into consideration the ultimate decision- 
making functions to be served , as well as the nature 
of the specific problem or situation unde:: analysis. 

The definition and assumptions stated above 
are closely related to the decision-making process, 
which leads to a consideration of the types of educa- 
tional decisions requiring evaluative information# 

The deve lo lent of a decision-oriented classification 
of the various types of evaluation has followed from 
this thinking. Five areas of evaluation may be 
identified : 

1. Systems Assessment — evaluations neces- 
sary to secure information concerning 
decisions relating to the state of the 
system. 

By the way, a sub-system assess- 
ment is possible* In this case, the 
charge would be to determine the present 
status of a specific objective and 
related objectives of a given instruc- 
tional program. 

2. Program Planning —those evaluations that 



will assist the decision maker to make 
planning decisions. He must have suffi- j 

cient information to be able to select j 

among alternative processes the one that 
will eventually accomplish his goal, | 

j 

This process takes place prior to j 

the implementation of the program. J 




l •*= 



3 * Program Implementation — this evaluation 
will tell us whether the program was 
introduced in the manner it was intended 
and to the group for which it was 
intended . 

4, Program Improvement — information secured 
as a result of evaluation that permits us 
to modify a program, if necessary* It is 
obtained during the course of a program 
and tells us how the program is function- 
ing, how enroute objectives are being 
achieved, and what unanticipated outcomes 
are being produced. 

5 . Program Certification — information 
secured as a result of evaluation that 
permits the decision maker to make judg- 
ments about the worth of the program and 
its potential generalizability to other 
related situations . 

According to the author, the evaluation areas 
outlined above seem to represent a growing consensus 
among a number of people engaged in the study of evalu- 
ation. 



I submit that the foregoing appears to pre- 
sent to us a workable, although sophisticated model for 
the development of evaluation in occupational programs. 
It appears to provide for process, product, varying 
admission levels and would appear to place our efforts 
in a favorable light as we further attack the problem 
of federal funding. What*s more, it has the option of 
either considering a total system or an individual 
program. 

Community College of Philadelphia 

At the Community College of Philadelphia 
prior to developing our 28 occupational curricula and 
options, we went through the process that is normally 
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accorded the introduction of new curricula. Community 
needs and student needs were assessed. Student interest 
was determined. Advisory committees were formed. 

Curricula were formulated. Positions for graduates were 
tentatively assured. Faculty were involved throughout 
the process. If I have missed a step , it is not because 
it was not done but simply because I have not listed it. 
This was the initial introduction of new occupational 
curricula. Obviously, as those experienced in program 
development know, continuous review must take place. 

In order to systematize this continuous review, wa 
devised a Career Curriculum Evaluation Form, which was 
to ba completed by concerned faculty under the leader- 
ship of our department heads and division directors. 

The goal of the form was to carefully analyze each 
given career curriculum and its specific elements. For 
example, we asked that the following headings be com- 
pleted: 

I . Name 

II. General objectives of curriculum 

III. Specific objectives of curriculum 

IV. Listing of courses by career specialty 
category, related or supporting course 
category, and general education category 

V. Relationship between each specific 

objective and specific courses in the 
curriculum. 

The intent of this task was to justify course 
inclusion based on a specific objective Important to 
the curriculum* The categorization of the courses was 
an attempt to create a taxonomy for thinking purposes • 

Other questions on the form included: 

VI, Job abilities or skills a graduate can 
perform upon graduation. 

VII. Job titles (entry positions) to which a 
graduate can aspire, to include salary 
ranges , 

, Selection criteria for entrance into program. 
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VIII 



Whan we were doing here was actually reassess 
ine the program based on some current input from our 
faculty. Many people refer to this as the continuous 
segment or element of curriculum development. Needless 
to say, some changes in the curricula were brought 
about as a result of this process. 



Our next project, which we hope to develop 
this year, is a follow-up study of our graduates and 
their employers. Working with interested faculty, we 
anticipate developing, testing and modifying a que. tion- 
naire and other techniques to be used in this follow 
up study. This next step will enable us to view what 
Dr. Medsker refers to as the "product. For, as he has 
implied and I do believe it, the "process method can 
not be separated from the "product" approach. 

One thing has struck me as I have analyzed 
this whole area of evaluation — what we are not doing 
is significant! And this definitely applies to my own 
situation • 



Current Evaluation Tgchnigugs 



The third part of my presentation is designed 
to briefly review existing evaluation techniques as 
reflected in the literature, 3 



Among them we have graduate interviews; 
career follow-up studies ; achievement testing; state 
and national licensing examination; industry advisdry 
committees; systems approach cycle; accreditation as 
an evaluation technique; self-initiated evaluation; 
personality change as a result of occupational education 



3 Arthur E. Bruhns, "Evaluation Processes 
Used to Access the Effectiveness of Vocational-Technical 
Programs," (Seminar Paper, UCLA, December 14, 1968), 
pp. 5-12, 



- 40 - 



44 



and evaluation; evaluative criteria; and cooperative 
education programs. I am sure that other techniques 
exist that have not been mentioned, but this does give 
us an idea of those aeveloped and ^n use. 



Conclusion 



X will close in saying that X agree with 
special report and recommendations by the Carnegie 
Commission on Higher Education of June 1970 that saidi 

The Carnegie Commission recommends a 
single program of institutional accred— 
itation for two-year colleges and the 
elimination of accreditation of 
specialities. The contribution of 
professional associations in the evalu- 
ation of specialized programs should be 
made through cooperation with the 
regional accrediting bodies.^ 



^The Open-Door Colleges . Clark Kerr, chairman 
(New York: McGraw-Hill Book Company, 1970), p. 49. 
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FACULTY EVALUATION: A PROJECT REPORT 



James L, Evanko 

Dean of Students, Allegheny Campus 
Community College of Allegheny County 



Why Evaluate 

Upon reviewing papers by McKeach^e , 

Boyer ,2 end Long, 3 one cen synthesize the reasons for 
faculty evaluation and define them in terms of three 
major objectives. 

The first objective is the improvement of 
teaching and the educational process. This is, in my 
opinion^ the most important. This objective can be 
evaluated in terms of the instructor s ability, 
course objectives and the classroom environment. 



^William J. McKeachie, "Student Ratings of 
j Faculty," American Associat i on of University Professors 

I Bulletin (Winter, 1969). 

\ I 

r 2 Marcia Boyer, "Teacher Evaluation: Toward 

| Improving Instructors," ERIC Junior College Research 

f Review , 4 (January, 1970), 

I 3 Louis Long, "Why Evaluate" (paper presented 

j at the 22nd National Conference on Higher Education, 

| Chicago, March 6, 1967). 

j 
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